CSEG3060 · Research Methodology in Computer Science
Critical appraisal, synthesis, gap identification and theoretical framing
Searching gives you a corpus. Analysis gives you an argument.
Unit III · Lecture 3 of 8 · 60 minutes (90 with activities) · CO2
Dr. Mohsin Furkh Dar · School of Computer Science, UPES
→ / Space next · ← back · O overview · N speaker notes · F full screen
Learning objectives · CO2
Critically appraise a paper using structured checklists — rigour, validity, contribution
Synthesise findings across studies using at least two formal techniques
Articulate research gaps using the canonical gap-statement grammar
Construct a theoretical framework linking theory to questions, propositions and method
Integrate all four techniques into a coherent literature review chapter
Orientation
Looks inward at each study: are the claims warranted by the evidence presented?
Looks outward across the corpus: what does the body of work say that no single paper says?
Looks at what is absent: a defensible gap that warrants new research.
Supplies the lens: the theory through which all of it is read and interpreted.
| Concept | One-line definition |
|---|---|
| Critical appraisal | A structured evaluation of the trustworthiness and value of a study |
| Synthesis | The integrative recombination of evidence across studies to produce new understanding |
| Research gap | An explicit, defensible absence in the body of knowledge that warrants new research |
| Theoretical framework | The conceptual scaffolding of theory guiding questions, hypotheses and interpretation |
Critical appraisal · 1 of 3
Were the methods appropriate, rigorously executed and transparently reported?
Is the theoretical grounding coherent, and are the constructs well defined?
Does the work advance knowledge beyond incremental description?
For each lens record both strengths and weaknesses, then tag the paper with a one-line verdict — e.g. “methodologically sound but conceptually narrow”. Forcing the verdict is what converts passive reading into appraisal.
In health research the randomised controlled trial sits at the top of the evidence hierarchy. In computing the strongest evidence may be a correctness proof or a benchmark on standardised datasets — so frameworks must be adapted, not imported.
Critical appraisal · 2 of 3
Sixteen domains for appraising reviews, not primary studies: protocol registration, comprehensiveness of the search, justification for excluding studies, risk-of-bias assessment, consideration of publication bias.
Critical appraisal · 3 of 3
| Validity type | Question posed | Typical CS threat |
|---|---|---|
| Internal | Did the treatment actually cause the effect? | Confounds, hyperparameter leakage, data snooping |
| Construct | Do the operationalisations measure the intended constructs? | Accuracy used as a proxy for “fairness” without justification |
| External | Can the findings generalise? | Single-dataset evaluation, no demographic diversity |
| Statistical conclusion | Are the statistical inferences correct? | Multiple comparisons uncorrected, underpowered studies |
“A Deep Learning Model for Phishing URL Detection Achieving 99.2% Accuracy”
Verdict: promising idea, but the evidence does not yet support the claimed performance level — treat as preliminary.
Synthesis · 1 of 2
Textual integration by theme, method or chronology. Flexible; exposed to reviewer bias.
Statistical aggregation of effect sizes. Most rigorous; needs comparable metrics.
Codes and themes developed then mapped across studies. Thomas & Harden (2008).
Combines empirical and theoretical sources. Suits conceptual and design-science work.
Uses a pre-existing framework as the organising scaffold.
f is a defensible inferential move — aggregation, contrast or explanation — not a mechanical combination. Naming the move is what makes the synthesis auditable.
Selection logic: comparable quantitative metrics → meta-analysis · heterogeneous or mixed evidence → thematic · an existing theory available as scaffold → framework synthesis.
Synthesis · 2 of 2
“Study A found…, Study B found…, Study C found…” — five sentences, zero claims.
Across the corpus, prediction accuracy is shaped more strongly by data characteristics — imbalance, distributional shift — than by algorithmic choice. While deep models show potential in data-rich settings, classical models retain a Pareto advantage when labelled data is scarce.
Gap identification · 1 of 2
| Gap type | What is absent | Design implication |
|---|---|---|
| Empirical | The phenomenon has not been examined, or not sufficiently | A study that measures it |
| Theoretical | Theory fails to explain the pattern, or none applied | A framework, or an application of one |
| Methodological | Prior work relies on a narrow set of methods | An alternative method or metric |
| Population / contextual | Findings restricted to one demographic, geography or domain | Evaluation in the missing context |
| Data | Required datasets absent, inaccessible or lacking attributes | A dataset contribution |
| Inconsistency | Conflicting results never reconciled | A replication or moderator analysis |
“No evaluation on low-resource Indic languages” → contextual
“Two reviews report contradictory effect sizes” → inconsistency
“Adoption theory never applied to QCaaS” → theoretical
Gap-by-omission — claiming a gap because you found no paper, rather than demonstrating absence through a structured search.
Gap identification · 2 of 2
“Despite [extensive/some] research on [A] in [context C], [little/no] attention has been paid to [B], particularly with respect to [specific dimension]. This is consequential because [reason]. Therefore, this study investigates [research question] using [method] in [context].”
“Despite extensive research on adversarial robustness in image classifiers, limited attention has been paid to the intersection of robustness and fairness, particularly for L2-bounded perturbations under realistic threat models. This intersection is consequential because deployed classifiers may fail unequally across subgroups, raising ethical and operational concerns. Therefore, this study investigates the fairness–robustness trade-off in medical imaging classifiers using a constrained optimisation framework.”
Theoretical framing · 1 of 2
| Aspect | Theoretical framework | Conceptual framework |
|---|---|---|
| Source | Established, citable theory (TAM, DOI) | Researcher-built diagram linking variables |
| Function | Supplies explanatory logic and propositions | Clarifies relationships to be tested |
| Testability | Generates falsifiable hypotheses | Provides a roadmap for analysis |
| Example | Technology Acceptance Model | A diagram linking code-review frequency, defect density and team size |
TAM · UTAUT · DOI · TOE · Activity Theory · Constructivism and Situated Learning · General Deterrence Theory · PDCA · Resource-Based View
Theory-of-the-month — adopting a theory because it is fashionable rather than because its constructs align with the research questions.
Practical application
Surface three to five candidates via seminal works, systematic reviews and citation tracing. Competition between candidates is what makes the final choice defensible.
Verify that the theory's constructs align with what you are actually asking; modify or extend where they do not.
Translate theoretical relationships into testable statements — e.g. H1: β1 > 0 for the effect of perceived usefulness on adoption intention.
State the boundary conditions under which the theory is believed to hold, so later disconfirmation is interpretable rather than fatal.
Theory: DOI combined with TOE
Constructs: relative advantage, compatibility, complexity (DOI); firm size, top-management support, regulatory environment, competitive pressure (TOE)
H1: relative advantage positively affects adoption
H2: top-management support moderates the effect of complexity on adoption
Assumptions: startups operate under resource scarcity; respondents are technically literate
The framework now determines the survey instrument, the sampling strategy, the analysis (PLS-SEM) and the interpretation — which is why framing precedes design.
Case study
23 primary studies appraised and tagged by dataset, quantisation method, hardware target and reported metrics. Studies flagged where latency, energy and accuracy are not reported jointly.
Three convergent themes: the quantisation-depth/accuracy trade-off (8-bit as Pareto sweet spot); hardware–software co-design outperforms algorithmic optimisation alone; workload heterogeneity — vision and speech diverge.
“Despite extensive research on quantisation and pruning for energy-efficient edge inference, limited attention has been paid to adaptive policies that dynamically trade accuracy for energy based on runtime context. This is consequential because static configurations underutilise the flexibility of modern edge accelerators. Therefore, this study proposes a reinforcement-learning-based adaptive inference framework and evaluates it on a Jetson Nano platform.”
The Markov Decision Process formalism supplies the lens; the policy is learned by deep RL. State, action and reward derive from system-level measurements.
Appraisal and synthesis without framing leaves questions empirically orphaned. Sophisticated theory on a poorly evidenced gap fails equally. Mastery requires all four.
Consolidation
| Pitfall | Description | Mitigation |
|---|---|---|
| Cataloguing | Listing studies without evaluation | Append appraisal and synthesis sentences to every cluster |
| Gap-by-omission | Claiming a gap because no paper was found | Demonstrate absence with a structured search |
| Theory-of-the-month | Adopting a fashionable theory | Justify selection by construct alignment |
| Synthesis-by-citation | Dropping names without integrating findings | Write integrative claims; cite evidence for and against |
| Mixing levels | Conflating empirical findings with opinion | Separate evidence from interpretation explicitly |
Diagnose your own draft: which of the five pitfalls does it currently exhibit?
Exam preparation
A senior researcher claims that “in fast-moving CS sub-fields, formal theoretical framing is impossible because theories become obsolete quickly.” Evaluate with reference to two theories.
Diagnose: “Across all surveyed studies, larger models always outperform smaller ones.” Name the synthesis technique, the embedded assumptions, and the conditions under which it fails.
Further reading: Booth, Sutton & Papaioannou (2016) · Kitchenham & Charters (2007) · Müller-Bloch & Kranz (2015) · Thomas & Harden (2008) · Webster & Watson (2002) · Wohlin (2014).